Appendix G — Practice Final Solutions

STAT303-2 (Winter2023)

Abstract

These solutions are being composed by students of the course STAT303-2 (Winter 2023).

G.1 Potential problems

Presence of which of the following potential problems in a linear regression model may lead to statistically significant variables appearing insignificant?

1. Multicollinearity

2. Outliers

3. Overfitting

Answer: A and B

Explanation:

A) Multicollinearity:

Recall, the estimated variance of the coefficient \(\beta_j\), of the \(j^{th}\) predictor \(X_j\), can be expressed as:

\[\hat{var}(\hat{\beta_j}) = \frac{(\hat{\sigma})^2}{(n-1)\hat{var}({X_j})}.\frac{1}{1-R^2_{X_j|X_{-j}}} \hspace{5cm} (1)\]

If the predictor \(X_j\) is collinear with other predictors, \(R^2_{X_j|X_{-j}}\) will be large, which in turn will inflate \(\hat{var}(\hat{\beta_j})\). In other words, multicollinearity inflates the standard errors of the coefficients for which the variables are collinear. Since \(t\)-statistic is calculated by dividing the estimated coefficient by its standard error, the \(t\)-statistics shrinks, and the corresponding \(p\)-value increases. Therefore, the hypothesis test loses the power to reject the null hypotheses, and thus statistically significant variables appearing insignificant.

Another way to think about this can be that if some predictors are collinear, it can be difficult to separate out the individual effects of these variables in the response and significant variables may appear insignificant.

B) Outliers

Recall, the estimate of error variance is given by:

\[\hat{\sigma}^2 = {\frac{RSS}{n-2}},\] where RSS is the residual sum of squared errors. Outliers result in an increase in \(RSS\), leading to an increase in the estimated error variance \(\hat{\sigma}^2\), which in turn inflates \(\hat{var}(\hat{\beta_j})\). The rest of the explanation follows from the previous explanation on multicollinearity.

C) Overfitting

Overfitting shrinks \(RSS\), which in turn shrinks \(\hat{\sigma}^2\), thereby shrinking \(\hat{var}(\hat{\beta_j})\). Thus overfitting will act in way opposite to what we observe in (A) and (B).

G.2 Potential problems

Classify a data point as influential / outlier / high leverage in a linear regression model, based on the description.

1. The data point is likely to have a large effect on the model in terms of prediction: Influential point

2. The data point has the potential to have a large effect on the model in terms of prediction: High leverage point

3. The data point is likely to inflate the model R-squared: High leverage point that is not influencial

4. The data point is unlikely to have a large effect on the model in terms of prediction: outlier

Explanation:

See the graphics in class presentation on Chapter3_Outliers_high_leverage_influential_points. Think of influential points / high leverage points / outliers as a force (proportional to the residual corresponding to the point) pulling a canteliver beam. Depending on the position from where you pull the cantilever beam, you may move it too much or too little.

A) Inluential point (high leverage & outlier): an outlier with the respect to both the predictor and the response. It has a large effect on the regression line. As shown in the graphics, influence is higher for more extreme outliers with same leverage and for points with higher leverage & similar outlying distance.

B) High leverage point: Observations with high leverage have an unusual value for the predictor (ie. lie outside the domain of most points). High leverage point has the potential to have a large affect on the regression line. It is cause for concern if the least squares line is heavily affected by just a couple of observations, because any problems with these points may invalidate the entire fit.

C) If you have a high leverage point that is not influencial: The variance of the response may increase in the presence of high leverage points, since an unusual set of predictor values may correspond to an unusual response, which may increase the total variation. However, as the point is not inluential, the increase in the unexplained variation (the squared residual) will not be proportionate to the increase in total variation. As \(R^2\) is one minus the ratio of unexplained variation to total variation, it is likely to increase.

D) Outliers: As shown in the graphics, outliers very small effect on prediction.

G.3 Autocorrelation

A linear regression model was developed to predict the number of passengers taking a flight per month. The data consists of number of passengers flying each month from January 1949 to December 1960. The autocorrelation plot below shows the correlation of the residuals with the lagged residuals of the model. Choose the most appropriate option.

1. The above plot shows the presence of autocorrelation. The 6-month lagged response is the most appropriate lag to be added as a predictor in the model to address autocorrelation

2. The above plot shows the presence of autocorrelation. The 12-month lagged response is the most appropriate lag to be added as a predictor in the model to address autocorrelation

3. The above plot shows the presence of autocorrelation. The 1-month lagged response is the most appropriate lag to be added as a predictor in the model to address autocorrelation

4. The above plot shows the absence of autocorrelation as the plot must have a cyclical pattern in the presence of autocorrelation

5. The above plot shows the absence of autocorrelation as the one month lagged residual must have the highest correlation with the residual in the presence of autocorrelation

Answer: B

Explanation: As seen in the plot, the residuals are highly correlated (correlation of more than 60%) with lagged residuals of 12 months. This shows the presence of autocorrelation. To address autocorrelation, the 12-month laggged response will be the most appropriate as it has the highest correlation with the response. Thus, it will explain the variation in the respone the most.

There is no need for there to be a cyclical pattern for autocorrelation. Even if one of the lagged residuals are highly correlated with the residual, it shows the presence of autocorrelation.

G.4 Logistic regression (goodness-of-fit)

Which of the following metrics can be used to assess the goodness-of-fit of a logistic regression model?

1. All of these

2. LL-Null

3. Log-Likelihood

4. Df Model

5. R-squared

Answer: Log-Likelihood

Explanation In logistic regression, the response is assumed to follow a Bernoulli distribution, where the probability of success is a function of the predictors and its coefficients (the model parameters). With this assumption, one can compute the the joint probability density of the observed data as a function of the model parameters. This creates a set of probability distributions (based on different values of model parameters) that could have generated the data. The algorithm finds the values of the model parameters (the beta coefficients) such that the probability of observing the data maximizes. This probability is the likelihood, and its logarithm is the log-likelihood. The higher the log-likelihood, the more probable it is to observe the data. Thus, log-likelihood is a way to measure the goodness-of-fit of the model.

LL-NULL is the log-likelihood of the model with no parameters. This is compared with the log-likelihood of the model with predictors to test if the regression is statistically significant.

Df Model is the number of predictors in the model.

R-squared cannot be used for logistic regression as there are no residuals.

G.5 Logistic regression (threshold probability)

For a logistic regression model, as we increase the decision threshold probability,

1. None of these

2. the recall will reduce or stay the same

3. the ROC-AUC will increase or stay the same

4. the precision will increase or stay the same

5. the classification accuracy will increase or stay the same

Answer: B

Explanation: See class slide on the confusion matrix below.

Increasing threshold probability means that less observations are predicted to be positive. Hence, some TP could turn into FN, reducing the recall. (this might not happen if there is no observations of actual positives between the thresholds). ROC-AUC is independent of the threshold probability. Both precision and classification accuracy might decrease if the number of FP among actual negatives increase more than the increase of TP among actual positives by the shift in the threshold.

G.6 Decision threshold probability

Which of the following metrics is independent of the decision threshold probability?

1. None of these

2. ROC-AUC

3. All of these (except the “None of these” option)

4. Precision

5. Recall

Answer: ROC-AUC

Explanation By changing the threshold, the number of points classified as negative and positive may change, and so TP, FP, TN and FN may change. Recall and precision may change as they are based on these metrics (TP ,FP, TN, and FN). However, the ROC-AUC specifically analyzes different thresholds. The ROC curve is a plot of TPR against FPR for all possible thresholds, and ROC-AUC is the area under the ROC curve, so the value itself is independent from the decision threshold probability.

G.7 Odds

Consider the following logistic regression model:

\[p(x) =\frac{1}{1+e^{-(\beta_0+\beta_1x)}}\].

Which of the following metrics will depend on the value of x?

1. Odds ratio when x increases by 2 units

2. increase in log odds when x increases by 10 units

3. All of these

4. Increase in predicted probability when x increases by 1 unit

E)none of these

Answer: D

Explanation:

\[p(x) =\frac{1}{1+e^{-(\beta_0+\beta_1x)}}\]

\[\implies \log\bigg(\frac{p(x)}{1-p(x)}\bigg) = \beta_0 + \beta_1x\]

\[\implies \log\big(Odds(x)\big) = \beta_0 + \beta_1x\]

When \(x\) increases by ‘c’ units,

\[p(x+c) - p(x) =\frac{1}{1+e^{-(\beta_0+\beta_1(x+c))}}-\frac{1}{1+e^{-(\beta_0+\beta_1(x))}}\]

\[log({Odds(x+c)}) - log({Odds(x)}) = \beta_1 c\]

\[\frac{Odds(x+c))}{Odds(x)} = e^{\beta_1 c}\]

We can see that only the increase in predicted probability when \(x\) increases by 1 unit is dependent on \(x\).

G.8 Precision-recall

We develop a logistic regression model to predict whether someone will pay a loan back or not. Loans are “approved” by us only for those borrowers who are predicted to pay back. The positive class is the borrowers that pay back the loans. What would a recall of 81% mean?

1. 81% of the borrowers that would pay back the loan are approved by us: Recall = TP/(TP + FN). TP here are those who are [approved by us] who [pay back the loan], while FN are those who were [not approved by us] but actually [pay back the loans]. The denominator is [all who pay back the loan]. Thus, Recall here means: among [all who pay back the loan], 81% are [approved by us].

2. Of all the loans we approve, 81% pay us back: This is Precision = TP/(TP + FP)

3. Of all the loans we don’t approve, 81% would not have paid us back if they were given the loan: This is the proportion of negatives correctly predicted - like precision for the negative class

4. Of all the loans we don’t approve, 19% would not have paid us back if they were given the loan: This is the proportion of negatives incorrectly predicted.

Answer: 81% of the borrowers that would pay back the loan are approved by us.

Explanation: Recall = True Positives/(True Positives + False Negatives).

In this case, True positives are those who got approved and would pay back. False Negatives are those we didn’t approve, but would pay back. Therefore, 81% Recall means 81% of the borrowers that would pay back the loan are approved by us.

G.9 Variable selection

Which of the following algorithms can be used for variable selection?

1. Lasso

2. Ridge regression

3. Forward stepwise selection

4. Best subset selection

Answer: A,C,D

Explanation: Both lasso and ridge regression are regularized least squares model, where the a shrinkage penalty is added to the ordinary least squares cost function. The shrinkage penalty in ridge regression shrinks the regression coefficients estimate towards zero, but not exactly zero, while the shrinkage penalty in lasso tends to give a set of zero regression coefficients and leads to a sparse model. Therefore, lasso can be used for variable selection, but not ridge regression.

Forward stepwise and best subset selection are variable selection algorithms by fitting multiple models having different combinations/number of predictors and choosing the best model.

G.10 Precision-recall

You are building a facial recognition model to allow people to unlock their phone. If the phone recognizes the person as the authorized user, it will unlock the phone. If it doesn’t recognize the user, it will prompt them to try again or try an alternative method (such as a passphrase). The facial recognition model is a classification model that identifies if the person unlocking the phone is the authorized user (positive response) or not (negative response).

Assume that letting a stranger (unauthorized user) unlock the phone is more risky (or more expensive) than not letting the authorized user unlock the phone.

Which of the following metric is the most important to optimize in the model?

1. Precision

2. Classification accuracy

3. Recall

4. ROC-AUC

Answer: Precision

Explanation:

A) Precision: Precision = TP/(TP + FN). Here, FN are those who are falsely assigned as an unauthorized user when they are actually the authorized user. FP are those who are assigned as the authorized user and are actually an unauthorized user. In this case, it’s important to optimize precision because it is more important to reduce the number of FP (strangers being recognized as the authorized user) than to reduce the number of FN (authorized user not being recognized).

B) Classification accuracy: This is incorrect because a model with high accuracy but a high FPR would be unacceptable since it would increase the risk of a stranger unlocking the phone.

C) Recall: This is incorrect because a high recall indicates that many of the positive cases are being detected. However, it does not measure the fraction of unauthorized users that the model identifies as authorized. A high FPR could lead to an unauthorized user unlocking the phone, which is a more expensive mistake than an FN.

D) ROC-AUC: This is incorrect because ROC-AUC does not take into account the cost of the positive and negative classes. It only measures how well the model can distinguish between authorized users and unauthorized users.

G.11 Logistic regression

Consider the following logistic regression model:

\[p(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_2 x_1 + \beta_2 x_2)}}\]

where assuming the threshold probability for classifying observations is 0.5. All observation with predicted probability greater than 0.5 are classified as belonging to class \(y=1\), while others are classified as belonging to class \(y=0\).

Which of the following plots correctly visualizes the predicted class based on \(x_1\) and \(x_2\)?

Answer: D

Explanation:

\(x_1\) will not have an impact on the outcome because its coefficient is 0. When \(x_2>5, p(x)\) will be less than \(0.5\) and \(y\) will equal 0, as the decisions threshold probability ois 0.5. When \(x_2<5, p(x)\) will be greater than \(0.5\) and \(y\) will equal \(1\).

G.12 ROC-AUC

In which of the following cases will ROC-AUC be the most appropriate metric to optimize among the all the performance metrics we have seen in this course.

1. There are wide disparities in the cost of false negatives vs. false positives, for example, predicting if the person has a serious disease.

2. The predicted probabilities will be used to rank observations, instead of classifying them, for example, the Google search engine using the predicted probabilities to rank pages in the decreasing order of relevance to the search query, instead of classifying the observations as ‘relevant’ and ‘not relevant’.

3. We wish to maximize the overall classification accuracy, for example, predicting if a person will vote for the Democrat or the Republican candidate in the US Presidential elections. Here, you may assume that the cost of false positives is similar to the cost of false negatives.

Answer: (B) only

Explanation:

(A) is incorrect because in cases where there are wide disparities in the cost of false negatives vs. false positives, it may be critical to minimize the performance metric associated with a higher loss. For example when predicting if the person has a serious disease, a false positive could lead to expensive and unnecessary medical treatment. Conversely, a false negative could result in a delay in diagnosis and treatment, potentially leading to a worse outcome. Thus, we want to prioritize minimizing false negatives. Since ROC-AUC is decision-threshold invariant, it’s not a useful metric for this type of optimization.

(B) is correct. ROC-AUC is scale-invariant. It measures how well predictions are ranked, rather than the absolute values of the predicted probabilities. Check the link.

(C) is incorrect because AUC is classification-threshold-invariant. It measures the quality of the model’s predictions irrespective of what classification threshold is chosen. However, the overall accuracy changes with change in decision threshold probability. To maximize overall accuracy, we need to find the optimal decision threshold probability.

G.13 Model selection

Which of the following linear model selection methods can be used when number of predictors is greater than the number of observations in linear regression?

1. Lasso

2. Ridge regression

3. Forward stepwise selection

4. Backward stepwise selection

5. Best subset selection

Answer: A, B, C

Expanation:

When number of predictors is greater than the number of observations, then, in case of ordinary least squares regression, the number of parameters are greater than the number of equations available to estimate those parameters. Thus, there is no unique solution. Also, in equation (1), \(R^2_{X_j|X_{-j}}\) is 1, and so \(\hat{var}(\hat{\beta_j})\) tends to infinity. Thus, it is not possible to fit an ordinary least squares model in this case. As backward stepwise selection begins with a model considering all predictors, it cannot be used as it is not possible to develop a model with all predictors in this case.

The best subset selection must consider models with all possible combination/number of predictors. Howvever, as number of predictors cannot be greater than the number of observations, we cannot develop all possible models, and thus we cannot use best subset selection.

Forward stepwise starts with no predictors and adds one predictor at a time. It is possible to keep adding predictors until the number of predictors (excluding the intercept) is one less than the number of observations. From this set of models, the best model can be chosen based on AIC, BIC, or any goodness-of-fit criteria that accounts for the number of predictors. Thus, it is possible to use forward stepwise selection.

The shrinkage penalty in lasso and ridge regression reduces the variance of the coefficents. The variance is infinity without any penalty when the number of predictors is greater than the number of observations. However, as lasso and ridge regression shrink the penalty, it is possible to obtain a unique solution.

The following is just for your information, but beyond the scope of this course: With lasso, there will be at most as many non-zero predictors as the number of observations. With ridge regression, there may be more non-zero predictors as compared to the number of observations.

G.14 Goodness-of-fit

G.14.1 Which of the following metrics can be used to compare the goodness-of-fit of models with different number of predictors?

1. AIC (Akaike Information criterion)

2. R-squared

3. Log-Likelihood

4. Pseudo R-squared

5. LLR p-value

Answer: AIC

Explanation: AIC is correct because it takes into account both the goodness-of-fit of the model and the complexity of the model (number of predictors).

\[AIC = -2logL + 2d,\]

where \(L\) is the maximized value of the likelihood function for the estimated model, and \(d\) is the number of predictors. From the above equation, we can see that \(AIC\) penalizes models with more parameters. Therefore, AIC allows for comparison between models with different numbers of predictors and helps to determine which model is the best fit.

All the other metrics will increase, while the LLR p-value will decrease with increase in number of predictors, and thus cannot be used to compare models with different number of predictors.

G.15 Model selection

Given a set of predictors, which of the following model selection methods guarantees to provide the best ‘least squares’ linear regression model, based on adjusted R-squared?

1. Best subset selection

2. Forward stepwise selection

3. Backward stepwise selection

4. Linear regression with all the statistically insignificant predictors removed

Answer: A

Explanation: Best subset selection considers every single model possible, while forward and backward selection don’t. Therefore, best subset selection necessarily gives the best model, while stepwise selection methods do not. For example, if a model consisting of predictors \(x_2\) and \(x_3\) is the best possible model, with regards to adjusted \(R\)-squared, while the model consisting of \(x_1\) is the best one predictor model, then forward stepwise selection will fail to identify the best possible model.

Adjusted \(R\)-squared depends on the residuals, among other things. However, the residuals don’t relate directly to statistical significance of predictors. Statistical significance of a predictor implies that the predictor is significantly linearly associated with the response, but it does not determine the variation in response explained by the predictor.

G.16 MSE estimate

Which of the following metrics gives the least biased estimate of MSE (mean squared error) on test data?

• Leave-one-out cross validation error
• MSE (mean squared error) on a test dataset (or validation set)
• K-fold cross validation error, where 1<k<n, where n is the number of observations
• All of these

Answer: Leave-one-out cross validation error

Explanation: Leave-one-out cross-validation offers two advantages:

- It provides a much less biased measure of test MSE compared to using a single test set because we repeatedly fit a model to a dataset that contains n-1 observations. - It tends not to overestimate the test MSE compared to using a single test set.

Textbook p200 Details: The test MSE gives us an idea of how well a model will perform on data it hasn’t previously seen, i.e. data that wasn’t used to “train” the model.

However, the drawback of using only one testing set is that the test MSE can vary greatly depending on which observations were used in the training and testing sets.

One way to avoid this problem is to fit a model several times using a different training and testing set each time, then calculating the test MSE to be the average of all of the test MSE’s.

Like the validation set approach, LOOCV involves splitting the set of observations into two parts (test & train). However, instead of creating two subsets of comparable size, LOOCV uses a single observation \((x_1,y_1)\) for the validation set, and the remaining observations \({(x_2, y_2), . . . , (x_n, y_n)}\) for the training set.

The statistical learning method is fit on the \(n − 1\) training observations, and a prediction \(\hat{y_1}\) is made for the excluded observation using its value \(x_1\). Since \((x_1,y_1)\) was not used in the fitting process, \(MSE_1 = (y_1 − \hat{y_1})^2\) provides an approximately unbiased estimate for the test error.

G.17 Question 17

Suppose you have a categorical predictor gender in the dataset with 3 distinct values - ‘male’, ‘female’, and ‘other’. Following are three ways to transform this predictor to make it suitable for forward stepwise selection. Which method is likely to provide the best model and which method is likely to provide the worst model, with regard to prediction accuracy on unknown (test) data?

1. Use the predictor gender as it is for forward stepwise selection.

2. Convert the predictor to 3 dummy variables - ‘male’, ‘female’, and ‘other’, where each dummy variable has 0s and 1s, depending on the ‘gender’, and use the dummy variables for forward stepwise selection, instead of gender

3. Replace the values of ‘male’ to 0, ‘female’ to 1, and ‘other’ to 2 in ‘gender’, and then use gender in forward stepwise selection

4. B is likely to provide the best model and, A or C are likely to provide the worst model

5. C is likely to provide the best model and A is likely to provide the worst model

6. C is likely to provide the best model and B is likely to provide the worst model

7. B is likely to provide the best model and A is likely to provide the worst model

8. None of these

Answer: B is likely to provide the best model, and A or C are likely to provide the worst model.

Explanation:

B is likely to provide the best model because breaking down a categorical variable into dummy variables allows us to perform stepwise selection for each class in gender. So it gives the algorithm the option to choose among more models than using the predictor gender as it is in the selection.

C is likely to provide a worse model as compared to B since it introduces an unreasonable constraint in the model that holding all other predictors constant, the difference in response between female and male is the same as the difference in response between other and female. In other words, we are converting categorical variables into ordinals which is constraining the model to find the relationship of the predictor in the order of male-female-other even though that may not be the case.

A is likely to provide a worse model as compared to B because it introuces a constraint in the model to either include all genders, or no gender. If only the female gender explains some variation in response, while there is no distinction between male and other, then the model should not be forced to keep the male and other categories. Suppose the other category happens to have a very few observations leading to an unstable coefficient (high standard error), then it may reduce the prediction accuracy.